Policy Optimization with Stochastic Mirror Descent

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes VRMPO algorithm: efficient policy gradient method with stochastic mirror descent. In VRMPO, novel variance-reduced estimator is presented to improve efficiency. We prove that the proposed needs only O(ε−3) trajectories achieve an ε-approximate first-order stationary point, which matches best complexity for optimization. Extensive empirical results demonstrate VRMP outperforms state-of-the-art methods various settings.